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WWC Review of the Report “Better Schools, Less Crime ?” 1 

The findings from this review do not reflect the full body of research evidence on school choice. 


What is this study about? 

The study examined the effect of school choice on 
the criminal activity, academic achievement, and 
high school graduation rate of more than 2,000 male 
middle and high school students in North Carolina’s 
Charlotte-Mecklenburg school district. 

For the 2002-03 school year, all district students were 
given the choice to either attend their neighborhood 
school or to apply to other schools in the district, 
where admission was not necessarily guaranteed. 

The intervention group consisted of male middle 
and high school students who applied to a non- 
neighborhood school that determined admission 
by a lottery and won admission to their first choice 
school. The comparison group consisted of similar 
students who lost a lottery for their first choice school. 

The study compared the academic achievement 
of the two groups 1 and 2 years after the lotteries 
were conducted, as well as the students’ criminal 
activity and high school graduation rates through 
2009, 7 years after the lotteries were conducted. 


Characteristics of Charlotte-Mecklenburg 
School Choice and the Study Sample 


Prior to the 2002-03 school year, the Charlotte- 
Mecklenberg school district switched from 
forced desegregation to a district-wide open 
enrollment policy. As a result, in the fall of 2002, 

60% of all students in the district enrolled in 
their neighborhood school, 30% applied to non- 
neighborhood schools where admission was either 
not possible or was guaranteed, and the remaining 
10% applied to non-neighborhood schools that 
had more applicants than available slots. These 
“over-subscribed” schools conducted lotteries to 
determine admission. 

This study focuses on male students in grades 6-1 1 
during the 2002-03 school year who had attended 
school in the same district the previous year and 
applied to non-neighborhood schools that used 
lotteries to determine admission. The intervention 
group consisted of the subset of these students 
who won the lottery and the comparison group 
consisted of those who did not. Those who did not 
win a lottery for their first choice school could attend 
their neighborhood school or a school further down 
their choice list. 
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What did the study find? 

Using student characteristics that have been shown 
to be highly correlated with future arrests, the author 
created a “risk index of future crime” for each stu- 
dent in the analysis sample and used it to divide 
students into low-risk and high-risk groups. 

Among all high-risk students, the author reported, 
and the WWC confirmed, that intervention group 
students committed crimes with lighter sentences 
than comparison group students. Additionally, high- 
risk high school intervention students were arrested 
for fewer felonies, and high-risk African-American 
intervention students were incarcerated for fewer 
days, than students with similar characteristics in the 
comparison group. There was no difference between 
the high-risk groups on academic achievement or 
high school graduation. 

Among students in the low-risk group, the study 
did not find any statistically significant differences 
between the intervention and comparison groups 
on criminal activity outcomes. The study author did 
not examine academic achievement and high school 
graduation for the low-risk group. 

In summary, the study author found that the intro- 
duction of school choice resulted in better outcomes 
related to criminal activity, but not academics, for 
middle and high school males who were at high risk 
for committing a future crime. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: The intervention and comparison 
groups were formed by a well-implemented random 
process. 

Cautions: The study had high levels of attrition 
for one outcome, the 2004 reading score. The 
study author demonstrated that students in the 
intervention and comparison groups were equivalent 
at baseline on reading achievement. Therefore, the 
analysis for this outcome meets WWC standards 
with reservations. 
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Appendix A: Study details 

Deming, D. J. (2011). Better schools, less crime? The Quarterly Journal of Economics, 126(4), 2063-2115. 


Setting 

The study used administrative data to examine the outcomes of students from North Caro- 
lina’s Charlotte-Mecklenburg school district from 2002-03 through 2008-09. 

Study sample 

The Charlotte-Mecklenburg school district implemented a district-wide open enrollment 
policy for the 2002-03 school year. Under this policy, students were guaranteed admission to 
their neighborhood school (independent of whether they had been previously attending that 
school), and 60% of the district’s students chose this option. Thirty percent of the students 
applied to non-neighborhood schools where admission was either not possible because the 
school was fully subscribed with neighborhood students or where admission was guaranteed 
because the number of applicants was fewer than the number of slots available. 

The study focused on the remaining 10% of students who applied to non-neighborhood 
schools that were oversubscribed. These schools held lotteries to determine which students 
would be admitted (the intervention group) and which would be denied admission (the com- 
parison group). The study only considered lotteries for students’ top school choice, known as 
“first-choice” schools. 

The study sample was further limited to males in grades 6-1 1 in 2002-03 who had been 
enrolled in a Charlotte-Mecklenburg school in the previous year. The analysis sample included 
a total of 2,095 students (1 ,01 4 high school and 1 ,081 middle school students) consisting of 
915 intervention students and 1,180 comparison students. However, the analytical sample 
sizes varied by outcome. 

The author used student characteristics that have been shown to be highly correlated with 
future arrests to define a risk index of future crime for each student who participated in a lot- 
tery. Based on this index, the author divided students into quintiles and analyzed data from 
low-risk students (the first through fourth quintiles) separately from data from high-risk stu- 
dents (the fifth quintile). The study analyzed outcomes for students in middle school lotteries 
and high school lotteries both together and separately. 

Intervention 

group 

The intervention group consisted of students who applied to attend an oversubscribed non- 
neighborhood school, won the selection lottery, and were admitted to their first-choice school. 

Comparison 

group 

The comparison group was made up of students who applied to attend an oversubscribed 
non-neighborhood school, lost the selection lottery, and were not admitted to their first-choice 
school. These students could have attended their neighborhood school or other schools fur- 
ther down on their choice list. 

Outcomes and 
measurement 

Criminal activity and high school graduation rates were measured in 2009, 7 years after the 
lotteries were conducted. Academic achievement was measured 1 and 2 years after the 
lotteries were conducted. For a more detailed description of these outcome measures, see 
Appendix B. 

Reason for 
review 

This study was identified for review by the WWC by receiving significant media attention. 
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Appendix B: Outcome measures for each domain 


Criminal activity 

Felony arrests 

The number of felonies committed by the average student. 

Sentence-weighted arrests 

The number of crimes committed by the average student, weighted by the crime's expected sentence. 

Total days incarcerated 

The number of days that the average student was incarcerated. 

Math achievement 

Math score (2003) 

The score from a standardized math exam for grades 3-8, expressed in standard deviation units. 

Math score (2004) 

The score from a standardized math exam for grades 3-8, expressed in standard deviation units. 

Reading achievement 

Reading score (2003) 

The score from a standardized reading exam for grades 3-8, expressed in standard deviation units. 

Reading score (2004) 

The score from a standardized reading exam for grades 3-8, expressed in standard deviation units. 

High school graduation 

High school graduation 

The proportion of students who graduated from a Charlotte— Mecklenburg school district high school. 


Table Notes: This single study review focuses on criminal activity outcomes, as they were the focus of the study. Within criminal activity outcomes, several are not included in the 
single study review for various reasons: all non-traffic arrests (because it includes misdemeanor crimes); property felonies, violent felonies, and drug felonies (because they are 
a subset of felony arrests); and total social cost (because it was defined subjectively). In addition to the criminal activity outcomes, the single study review includes key academic 
outcomes that were presented in supplementary analyses in the study: math and reading achievement and high school graduation rate. 
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Appendix C: Study findings for each domain 




Mean 

(standard deviation) 


WWC calculations 


Domain and Study 

outcome measure sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Criminal activity, middle and high school students, high-risk group 

Felony arrests Middle and 

high school 
students, 
high-risk 
group 

418 

students 

0.59 

(1.36) 

0.73 

(1.36) 

-0.14 

-0.10 

-4 

>0.05 

Sentence-weighted arrests Middle and 

high school 
students, 
high-risk 
group 

418 

students 

26.60 

(116.06) 

52.50 

(171.52) 

-25.90 

-0.17 

-7 

<0.05 

Domain average for criminal activity, middle and high school students, high-risk group 

-0.14 

-6 

Statistically 

significant 

Criminal activity, high school students, high-risk group 

Felony arrests High school 

students, 

high-risk 

group 

202 

students 

0.43 

(1.18) 

0.77 

(1,43) 

-0.34 

-0.26 

-10 

<0.01 

Sentence-weighted arrests High school 

students, 

high-risk 

group 

202 

students 

35.50 

(99.70) 

58.60 

(192.10) 

-23.10 

-0.15 

-6 

>0.05 

Domain average for criminal activity, high school students, high-risk group 


-0.21 

-8 

Statistically 

significant 

Criminal activity, middle school students, high-risk group 

Felony arrests Middle school 

students, 

high-risk 

group 

216 

students 

0.78 

(1.54) 

0.71 

(1.32) 

0.07 

0.05 

2 

>0.05 

Sentence-weighted arrests Middle school 

students, 

high-risk 

group 

216 

students 

17.30 

(135.20) 

48.30 

(156.40) 

-31.00 

-0.21 

-8 

<0.05 

Domain average for criminal activity, middle school students, high-risk group 


-0.08 

-3 

Statistically 

significant 

Criminal activity, middle and high school students, 

low-risk group 






Felony arrests Middle and 

high school 
students, 
low-risk 
group 

1,677 

students 

0.11 

(0.50) 

0.10 

(0.48) 

0.01 

0.03 

+1 

>0.05 

Sentence-weighted arrests Middle and 

high school 
students, 
low-risk 
group 

1,677 

students 

6.90 

(45.59) 

3.80 

(31.46) 

3.10 

0.08 

+3 

>0.05 
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Domain average for criminal activity, middle and high schools students 

, low-risk group 


0.06 

+2 

Not 

statistically 

significant 

Math achievement, middle school students, high-risk group 

Math score (2003) Middle school 171 -0.98 

students, students (1.00) 

high-risk 
group 

-1.03 

(1.00) 

0.05 

0.05 

+2 

>0.05 

Math score (2004) Middle school 137 -1.02 

students, students (1.00) 

high-risk 
group 

-0.93 

(1.00) 

-0.09 

-0.09 

-4 

>0.05 

Domain average for math achievement, middle school students, high-risk group 


-0.02 

-1 

Not 

statistically 

significant 

Reading achievement, middle school students, high-risk group 

Reading score (2003) Middle school 172 -1.24 

students, students (1.00) 

high-risk 
group 

-1.16 

(1.00) 

-0.08 

-0.08 

-3 

>0.05 

Reading score (2004) Middle school 135 1.27 

students, students (1.00) 

high-risk 
group 

-1.19 

(1.00) 

-0.08 

-0.08 

-3 

>0.05 

Domain average for reading achievement, middle school students, high-risk group 


-0.08 

-3 

Not 

statistically 

significant 

High school graduation, middle school students, high-risk group 

High school graduation Middle school 216 0.08 

students, students (0.26) 

high-risk 
group 

0.11 

(0.31) 

-0.03 

-0.11 

-4 

>0.05 

Domain average for high school graduation, middle school students, high-risk group 


-0.11 

-4 

Not 

statistically 

significant 

High school graduation, high school students, high-risk group 

High school graduation High school 202 0.24 

students, students (0.43) 

high-risk 
group 

0.27 

(0.44) 

-0.03 

-0.07 

-3 

>0.05 

Domain average for high school graduation, high school students, high-risk group 


-0.07 

-3 

Not 

statistically 

significant 


Table Notes: For the criminal activity domain, negative results for mean difference, effect size, and improvement index favor the intervention group, and positive results favor the 
comparison group. For all other domains, positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the compari- 
son group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an aver- 
age student's outcome that can be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average student’s percentile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC; 
the study is characterized as having a statistically significant effect on reducing criminal activity of high-risk high school and middle school students because univariate statistical 
tests are reported for each outcome measure, the effect for at least one measure within each domain is positive and statistically significant, and no effects are negative and statis- 
tically significant. The study is characterized as having no discernible effects on math achievement, reading achievement, and high school graduation rates, because the study did 
not show a statistically significant or substantively important effect on outcomes in those domains. Throughout the table, the term “middle school student” refers to students who 
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participated in lotteries for middle school attendance, and “high school student” refers to students who participated in lotteries for high school attendance. Since some outcomes 
were analyzed 7 years after the lotteries, the terms do not always reflect the school of attendance at the time the outcomes were measured. 

Study Notes: No corrections for clustering or multiple comparisons were needed. The p-values presented here were reported in the original study. The comparison group mean 
and the mean differences between intervention and comparison groups are as reported in the study. The intervention group mean was calculated by the WWC as the comparison 
group mean plus the mean difference. Effect sizes and improvement indices were calculated using standard deviations provided to the WWC by the study author. Analyses on math 
achievement, reading achievement, and high school graduation rates were only conducted for high-risk groups. 
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Appendix D: Supplemental findings by domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Criminal activity, African-American subgroups 

Total days incarcerated 

African- 
American 
middle and 
high school 
students, 
high-risk 
group 

395 

students 

40.10 

(167.95) 

70.00 

(193.81) 

-29.90 

-0.16 

-7 

<0.01 

Total days incarcerated 

African- 
American 
high school 
students, 
high-risk 
group 

190 

students 

64.70 

(217.40) 

91.40 

(253.40) 

-26.70 

-0.11 

-5 

>0.05 

Total days incarcerated 

African- 

American 

middle 

school 

students, 

high-risk 

group 

205 

students 

19.30 

(165.50) 

55.50 

(140.40) 

-36.20 

-0.24 

-9 

<0.01 

Total days incarcerated 

African- 
American 
middle and 
high school 
students, 
low-risk 
group 

864 

students 

13.00 

(64.86) 

780 

(51.37) 

5.20 

0.09 

+4 

>0.05 


Table Notes: For the criminal activity domain, negative results for mean difference, effect size, and improvement index favor the intervention group, and positive results favor the 
comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in 
an average student's outcome that can be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. Throughout the table, the term “middle school student” refers to stu- 
dents who participated in lotteries for middle school attendance, and “high school student” refers to students who participated in lotteries for high school attendance. Since some 
outcomes were analyzed 7 years after the lotteries, the terms do not always reflect the school of attendance at the time the outcomes were measured. 

Study Notes: No corrections for clustering or multiple comparisons were needed. The p-values presented here were reported in the original study. The comparison group mean 
and the mean differences between intervention and comparison groups are as reported in the study. The intervention group mean was calculated by the WWC as the comparison 
group mean plus the mean difference. Effect sizes and improvement indices were calculated using standard deviations provided to the WWC by the study author. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. A quick review of this study was released 
on March 8, 2012, and this report is the follow-up review that replaces that initial assessment. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, July). WWC 
review of the report: Better schools, less crime? Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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